Skip navigation and go to content

Snapshot Testing the Accessibility Tree

On this page

We are going to write a test that creates a snapshot of the Accessibility Tree and isolates the header element.

Since screen reader output is not something we can assert programmatically quite yet, snapshot testing techniques for the Accessibility Tree can help us approximate it.

Also, limiting the scope of the test to a smaller part of the Accessibility Tree will help make it less brittle. If something changes elsewhere on a page, it will be less likely that you’ll have to rewrite all your snapshot tests.

💡Tip

Isolating parts of the Accessibility Tree for snapshot testing can make your tests more useful and less brittle.

There is a script already set up in package.json that will start Puppeteer via jest-puppeteer.

yarn test:puppeteer

Adding the --watch flag will re-run tests as the file is saved.

There is also a jest-puppeteer.config.js file that is used to start up the local dev server and launch Puppeteer’s Chromium browser instance with support for experimental web platform features, but without the sandbox.

💡Tip

The no-sandbox flag is not something I necessarily like from a security perspective, but it does allow jest-puppeteer to work on Windows. If you are a Windows user, it may be worth digging into this issue further.

module.exports = {
    server: {
      command: 'parcel index.html'
    },
    launch: {
      args: ['--no-sandbox', '--enable-experimental-web-platform-features'],
    }
}
Video: Project Overview
Loaded: 7%
Current Time 0:00
/
Duration Time 1:10
Video Transcript

So the idea that we're gonna work through or look through today is how do you use Puppeteer to get an accessibility snapshot of the accessibility tree? So with jest-puppeteer, that's the tool that we are going to use, in the ReadMe, we've got some details on how to run it.

So we have a script in package.json with yarn test:puppeteer. Let's go check that out in package.json. So our test command for test:puppeteer will run jest, and it's pointing it at the exercise4.puppeteer directory. Any files using a wildcard selector within that directory. We have a jest-puppeteer.config that is running our parcel command.

So starting a starting parcel for our index page, and it's also passing some arguments here when it launches Chrome, including no sandbox and experimental web platform features. The no-sandbox thing is really only I've had it come up on Windows. So this is working on Windows now, I'm happy to say, by using this no-sandbox flag, which I don't love from a security perspective, but this is how I could get this to work on Windows.

Writing the First Test

Inside of the exercise4-puppeteer directory is

The first thing to do at the top of the file is tell Jest we want to use the Puppeteer environment as opposed to JSDOM.

We’ll import expect-puppeteer as well as path and fs from the Node.js filesystem APIs:

/**
/* @jest-environment puppeteer
**/

import 'expect-puppeteer'
import path from 'path'
import fs from 'fs'

As we’ve seen in other test files we’ve written, we’ll write a describe statement and a beforeAll block inside to tell Puppeteer to navigate to the Passes page on the local dev server.

This time it’s a little different as the beforeAll callback function will be asynchronous using async/await. Put the async keyword at the start of the callback function to invoke this behavior. Inside of the function body, we’ll await a call to page.goto the local development server.

The first test we’ll write will be an it block that says that the “Pick a Plan and Start Your Adventure Today!” heading text should appear somewhere on the page. This will also have an async function as the callback.

Referencing the expect-puppeteer docs, we can use toMatch to say we expect text to appear on the page.

Here’s what it looks like all together at this point:

import 'expect-puppeteer'
import path from 'path'
import fs from 'fs'

describe('Accessibility Tree', () => {
    beforeAll(async () => {
        await page.goto('http://localhost:1234/passes')
    })

    it('should display heading text on page', async () => {
        await expect(page).toMatch('Pick a Plan and Start Your Adventure Today!')
    })
})

Running the Puppeteer tests with yarn test:puppeteer, we can see this test passes!

Jest shows our heading test passes.

Writing the Tree Snapshot Test

Now let’s write a test that writes a snapshot of the Accessibility Tree.

To create the snapshot, we’ll call page.accessibility.snapshot(). There are some options that can be passed in. We’ll use them later.

To save the file, first create a variable for the file path by calling path.join(process.cwd()) to get the current working directory. Further arguments to path.join will create subdirectories and the last arg will be the filename to write.

From there, we’ll use fs.writeFile and pass in the file path to the exercise directory and the stringified JSON output of the snapshot.

it('should create a JSON file for the accessibility tree', async() => {
        const snapshot = await page.accessibility.snapshot()

        const assetFilePath = path.join(process.cwd(), 'exercise4-puppeteer', 'a11y-tree.json')
        fs.writeFile(
            assetFilePath,
            JSON.stringify(snapshot, null, 5), 
            () => {}
        )
 })

Because this is a test file, we should include an actual test.

We will write an assertion that the file was created, even though we can see it show up in the sidebar when we run the test command.

Using the fs.stat() Node filesystem API, we can pass in the file path we created and inside of the callback add an assertion that the file was successfully created:

fs.stat(assetFilePath, (err, stat) => {
	expect(stat.isFile()).toBe(true)
})

Once we run the test command, the a11y-tree.json file should be created.

Video: Writing the Accessibility Tree Snapshot Test
Loaded: 2%
Current Time 0:00
/
Duration Time 3:22
Video Transcript

And so since I already had a snapshot, our test is failing. Let's go see what it looks like. So we've got this accessibility tree test. Let's go look at this. So I've got a test written to describe the accessibility tree.

We tell puppeteer using async await. We can say page.goto and go tell it to go to any page. And we can set up snapshots for all of our pages. And it does a little bit of a generic test just to check and make sure that this is on the right page. So checking that it should have the heading of "Pick a Plan and Start Your Adventure Today!" So using the puppeteer API for that, and then we get into some specific things about the accessibility tree.

So using puppeteers page.accessibility.snapshot API and it has this option that you can pass to it, it says interesting, only, true. That changes the amount of information that gets put into the JSON schema, the file that, 'cuz we're generating a file with this result. So we can actually go and view it, and I'll show it to you in a second.

But interesting only will change what gets output into that JSON file. So I'm using node here to write that to a file. It's gonna overwrite every time if that file already exists. It is something we can store in source control. The snapshot is slightly different. I'm creating a JSON file just so I can go see what's in it. But the snapshot, those get put in this snapshots directory. So this is actually the thing that is capturing our snapshot and pointing out what changed. So we've got a file we can go and inspect.

Here's the test that was failing. So it has the correct accessibility tree. So this is writing an assertion to expect that the snapshot that we had in source control matches the one that just ran when I ran the project and there was some change in the page. We saw there was changes to buttons not being wrapped with headings anymore. We saw a change with image alt text potentially and a page title. So I can go and review those.

Open our terminal back in here again, 'cuz the snapshot didn't match. So we had some differences. So this gives you an opportunity to go and review some of this underlying accessibility API information to see if things changed. With this tool I'd say it's a little bit limited in its utility just because, I don't know, I think functionally, we can have a lot more robustness with less friction checking things like labels and titles. And that's checking a lot of the same stuff that we could check with Cypress or with Jest, but maybe there is some utility in having a snapshot of everything. I just think it's really cool that you can do this. So I wanted to show it to you.

Let's look at what these outputted files look like. Ally tree or a11y-tree.json is all of the children. So the role that root web area, the name of the page is the page title. So that is something that changed in our page passes. Come over here in an actual page. This react helmet, when I added this title, that changed from what was in the snapshot before. That's a good change. That's a title that's more specific, that's not just "Camp Spots" which if every page says "Camp Spots" the title's not as useful anymore. So that's a good change.

Writing a Snapshot Comparison Test

Inside of a11y-tree.test.js, add another it block where we will test if the current page snapshot matches a previous snapshot.

We’ll create a variable for the called axTree that will take a snapshot of the current page. Then we’ll add an expect that checks it against a saved a11y-tree.test.js.snap file inside the __snapshots__ directory.

it(`has the correct axTree`, async () => {
	const axTree = await page.accessibility.snapshot()
	expect(axTree).toMatchSnapshot()
})

When the test runs it fails, as the older snapshot is out of date.

Jest shows a test failure because the snapshots don’t match.Loading
Video: Examining a Snapshot
Loaded: 4%
Current Time 0:00
/
Duration Time 1:51
Video Transcript

Looking at the snap, the jest snapshot. So this is where it was coming up with kind of a diff of what changed. And that format is being created to be machine readable. So really this terminal output is what is useful. So coming down here and just seeing, are these changes good? Like what's going on with the MegaNav. It's saying role of button, whereas those were wrapped in headings before.

Before I generate a new snapshot and just give it the okay, I'm gonna go and check what's going on with these. I think CampSpots image. We made that change to remove the alt text from or to add an empty alt attribute to an image. So I think that part should be fine still but I could just open this up in a browser and come up here and look.

So if I'm wanting to make sure what's the state of things, I think our image, it is probably missing that alt attribute. Maybe I blew that change. Oh, I think this might be an old server. This isn't showing our changes. But I can come in here and come and look at some of the stuff like our megamenu. Yeah, we haven't. This is the version that hasn't been fixed yet. What I might do is just run a server, do a little check, compare.

I've got the snapshot telling me what it found but as I'm trying to make sense of that it can help to come and look at, okay, what did it see? Is this stuff okay? Is this a good change? So here's our empty alt attribute. That's a good thing. 'Cause that's, should be our CampSpots image. And then coming over here to our MegaNav, we have buttons inside of h2s. It looks good to me. That's the way it's supposed to be. These headings add heading structure to the page. The buttons are the interactive parts inside of those. That's a good change.

But we don’t want to fix this test by undoing all of our work on the site’s markup!

Instead, at the bottom of the error message Jest instructs us to add a -u flag to the test command to update the saved snapshot:

Jest instructs us to add -u to update the snapshot.Loading

Running the command and updating the snapshot results in a passing test:

The test passes after updating the snapshotLoading
Video: Updating a Snapshot
Loaded: 6%
Current Time 0:00
/
Duration Time 1:16
Video Transcript

So I'm gonna give the sign off for the snapshot. So when I'm in here for yarn test puppeteer, and I can do dash watch, if I wanna interactively check out these changes. So it's running just like our unit tests are, but it's the specific puppeteer process, of running headless Chrome.

And so it said one snapshot failed. We kind of went through and evaluated everything. And I can come in here, and it says, "Inspect your code changes or press U to update them."

So we did inspect the code changes, I'm gonna hit update, and it will update that snapshot, and it got that test to pass. And so now, in my get environment over here, we'll hit refresh, and make sure, so this snap file, this now can be checked into source control, and I can go and see, cool, that's good.

So then the next time my colleague comes in, or something, and they run this test, if something changed, forces them to come in, and double check the accessibility output. So that's pretty cool. The fact that we can automate stuff about the accessibility tree, has a lot of promise. It's a cool part of automation, since we can't really automate assistive technology.

Making Snapshots More Interesting

The a11y-tree.json file resulting from running the test looks like this:

{
     "role": "RootWebArea",
     "name": "CampSpots Passes",
     "children": [
          {
               "role": "heading",
               "name": "CampSpots",
               "level": 1
          },
          {
               "role": "link",
               "name": "CampSpots"
          },
          {
               "role": "heading",
               "name": "Plan Your Trip",
               "level": 2
          },
					...

To get more information, we can pass a config object into page.accessibility.snapshot() that sets the option to be false:

const snapshot = await page.accessibility.snapshot({
	interestingOnly: false
})

Now when re-running the test, the JSON output includes a lot more information than it did before:

{
  "role": "RootWebArea",
  "name": "CampSpots Passes",
  "children": [
    {
      "role": "generic",
      "name": "",
      "children": [
        {
          "role": "generic",
          "name": "",
          "children": [
            {
              "role": "generic",
              "name": "",
              "children": [
                {
                  "role": "heading",
                  "name": "CampSpots",
                  "level": 1,
                  "children": [
                    {
                      "role": "StaticText",
                      "name": "CampSpots"
                    }
                  ]
                }
              ]

Almost by definition, “generic” is not interesting, but it is part of the structure. It’s useful to look at the “before and after” of using the interestingOnly option. Depending on the test you’re writing, it might be a better idea to only capture the more pertinent information.

Final Thoughts on jest-puppeteer

In most cases, I would suggest sticking with Cypress and Jest for the types of tests we’ve been working with so far. However, there might be a situation in your work where you would benefit from being able to test against an entire page snapshot.